Pull/Run the docker image containing molecule and friends, Mount the playbooks and roles repository, Run the desired test suite (handled by ansible/tests/detox_wrapper.sh)
Molecule test Ansible
Mons, 2019-03-21
About the author
User & contributor in Molecule / Ansible ecosystem.
Free Software Infrastructure Automation.
Culture, Automation, Measurement, Sharing.
Ansible User stories
Day 2 routines: system patches, audit, inventory.
Reproductible provisioning, from hypervisor to apps.
Automated backup & restore data.
Maintain environments on shared hosting platforms.
Deploy software.
Build CI pipelines.
Manage everything API: network, cloud, kubernetes.
Common problem
Validate roles and playbooks before production
Instantiate temporary infrastructure
Solution
Disclaimer
| Best practices to get further: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html |
Tooling lanscape
Molecule has many friends in the toolbox.
Ansible ecosystem
Platforms backends
Dependency backends
Verifiers
Ansible
Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates. Ansible’s main goals are simplicity and ease-of-use.
— https://docs.ansible.com/ansible
Ansible Installation
Requirements: Python 2.7 or >= 3.6
sudo apt-get install -y python-pip libssl-dev
pip install --user ansibleMolecule
Molecule is designed to aid in the development and testing of Ansible roles. […] Molecule is opinionated in order to encourage an approach that results in consistently developed roles that are well-written, easily understood and maintained.
— https://molecule.readthedocs.io
Molecule Installation
Requirements: Ansible
sudo apt-get install -y python-pip libssl-dev
pip install --user moleculeAnsible-lint
Improve the roles quality.
Kill opinion wars.
Installed as a molecule dependency.
Ansible Lint is a commandline tool for linting playbooks. Use it to detect behaviors and practices that could potentially be improved.
— https://docs.ansible.com/ansible-lint
Backends
(local) Virtualization
Cloud provider
Bake your own
(local) Virtualization
Docker
LXC
LXD
Vagrant
Cloud provider
Azure
EC2
GCE
Linode
Openstack
| Slow! Keep it for specific cloud features, Windows. |
Bake your own
Delegated
Verifier
Audit the state of the tested platform after role execution with an independant tool.
Testinfra
Goss
Inspec
Testinfra
Default verifier.
Write tests in python.
Public == developers.
Goss
Easy. YAML syntax, fit well in the Ansible ecosystem.
Fast. Near instantaneous.
Small. <10MB single self-contained binary.
Linux only.
Goss is a YAML based serverspec alternative tool for validating a server’s configuration.
— https://goss.rocks
Inspec
Complex, ruby based, with a feature full DSL.
Linux, MacOS and Windows support.
InSpec is compliance as code. Turn your compliance, security, and other policy requirements into automated tests.
— https://www.inspec.io/
Molecule scenario dissected
$ molecule matrix -s default test
--> Test matrix
└── default
├── lint
├── destroy
├── dependency
├── syntax
├── create
├── prepare
├── converge
├── idempotence
├── side_effect
├── verify
└── destroylint
Enforce syntax rules (Ansible-lint, Yamllint).
Kill opinion wars and improve the roles quality.
--> Executing Ansible Lint on molecule/default/playbook.yml...
[701] No 'galaxy_info' found
meta/main.yml:1
[306] Shells that use pipes should set the pipefail option
molecule/default/playbook.yml:20
Task/Handler: shell | get version of common_linux
[206] Variables should have spaces before and after: {{ var_name }}
tasks/task_60_cron.yml:19
path: "/etc/cron.{{cron_item}}"destroy
Destroy the temporary platforms.
molecule destroy
--> Action: 'destroy'
PLAY [Destroy] *****************************************************************
TASK [Destroy molecule instance(s)] ********************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Wait for instance(s) deletion to complete] *******************************
ok: [localhost] => (item=None)
ok: [localhost]
TASK [Delete docker network(s)] ************************************************
PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0dependency
Ensure the external requirements are installed
Supported backends:
galaxy
gilt
shell
--> Action: 'dependency'
- downloading role 'repo-remi', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-repo-remi/archive/1.2.0.tar.gz
- extracting geerlingguy.repo-remi to /tmp/molecule/ansible-role-phpmyadmin/default/roles/geerlingguy.repo-remi
- geerlingguy.repo-remi (1.2.0) was installed successfully
- downloading role 'apache', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/3.0.3.tar.gz
- extracting geerlingguy.apache to /tmp/molecule/ansible-role-phpmyadmin/default/roles/geerlingguy.apache
- geerlingguy.apache (3.0.3) was installed successfully
- downloading role 'mysql', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-mysql/archive/2.9.4.tar.gz
- extracting geerlingguy.mysql to /tmp/molecule/ansible-role-phpmyadmin/default/roles/geerlingguy.mysql
- geerlingguy.mysql (2.9.4) was installed successfully
- downloading role 'php-versions', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-php-versions/archive/3.0.0.tar.gz
- extracting geerlingguy.php-versions to /tmp/molecule/ansible-role-phpmyadmin/default/roles/geerlingguy.php-versions
- geerlingguy.php-versions (3.0.0) was installed successfully
- downloading role 'php', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-php/archive/3.7.0.tar.gz
- extracting geerlingguy.php to /tmp/molecule/ansible-role-phpmyadmin/default/roles/geerlingguy.php
- geerlingguy.php (3.7.0) was installed successfully
- downloading role 'php-mysql', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-php-mysql/archive/2.0.2.tar.gz
- extracting geerlingguy.php-mysql to /tmp/molecule/ansible-role-phpmyadmin/default/roles/geerlingguy.php-mysql
- geerlingguy.php-mysql (2.0.2) was installed successfullysyntax
Complementary to lint.
Perform a syntax check on the playbook.
ansible-playbook --syntax-checkmolecule syntax
--> Validating schema molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix
└── default
└── syntax
--> Scenario: 'default'
--> Action: 'syntax'
playbook: molecule/default/playbook.ymlcreate
Create the temporary platforms needed to execute the tests.
Faster and more reliable results are achieved with local drivers.
Remote drivers because are slow and error prone.
--> Scenario: 'default'
--> Action: 'create'
PLAY [Create] ******************************************************************
TASK [Log into a Docker registry] **********************************************
skipping: [localhost] => (item=None)
TASK [Create Dockerfiles from image names] *************************************
skipping: [localhost] => (item=None)
TASK [Discover local Docker images] ********************************************
ok: [localhost] => (item=None)
ok: [localhost]
TASK [Build an Ansible compatible image] ***************************************
skipping: [localhost] => (item=None)
TASK [Create docker network(s)] ************************************************
TASK [Create molecule instance(s)] *********************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Wait for instance(s) creation to complete] *******************************
FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (299 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (298 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (297 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (296 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (295 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (294 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (293 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (292 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (291 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (290 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (289 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (288 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (287 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (286 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (285 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (284 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (283 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (282 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (281 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (280 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (279 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (278 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (277 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (276 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (275 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (274 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (273 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (272 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (271 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (270 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (269 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (268 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (267 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (266 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (265 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (264 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (263 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (262 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (261 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (260 retries left).
changed: [localhost] => (item=None)
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=3 changed=2 unreachable=0 failed=0prepare
Optional.
Executes actions which bring the system to a given state prior to converge.
--> Scenario: 'default'
--> Action: 'prepare'
Skipping, prepare playbook not configured.converge
Execute the main role.
--> Scenario: 'default'
--> Action: 'converge'
PLAY [Converge] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [instance]
TASK [Update apt cache.] *******************************************************
skipping: [instance]
TASK [geerlingguy.repo-remi : Install remi repo.] ******************************
changed: [instance]
TASK [geerlingguy.repo-remi : Import remi GPG key.] ****************************
changed: [instance]
TASK [geerlingguy.apache : Include OS-specific variables.] *********************
ok: [instance]
TASK [geerlingguy.apache : Include variables for Amazon Linux.] ****************
skipping: [instance]
TASK [geerlingguy.apache : Define apache_packages.] ****************************
ok: [instance]
[...]idempotence
The idempotence sequence is key
The main role is executed again; the result should change nothing in order to achieve idempotence.
Idempotence is a goal difficult to achieve, particularly on Windows.
side_effect
Optional
Executes actions which are not in the role, after converge.
--> Scenario: 'default'
--> Action: 'side_effect'
Skipping, side_effect playbook not configured.verify
Executing an audit tool to verify that the final state is meeting expectations.
Supported backends:
testinfra - default, python based.
Goss - Linux only. Easy and fast.
Inspec - ruby DSL, works well for Windows targets.
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/fab/src/themr0c/talk-jdl2019/examples/geerlingguy.phpmyadmin/molecule/default/tests/...
============================= test session starts ==============================
platform linux2 -- Python 2.7.16, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/fab/src/themr0c/talk-jdl2019/examples/geerlingguy.phpmyadmin/molecule/default, inifile:
plugins: testinfra-1.16.0
collected 1 item
tests/test_default.py . [100%]
========================== 1 passed in 12.76 seconds ===========================
Verifier completed successfully.destroy
Destroy the temporary platforms.
molecule destroy
--> Action: 'destroy'
PLAY [Destroy] *****************************************************************
TASK [Destroy molecule instance(s)] ********************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Wait for instance(s) deletion to complete] *******************************
ok: [localhost] => (item=None)
ok: [localhost]
TASK [Delete docker network(s)] ************************************************
PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0From molecule to delivery pipeline
Objective: validate that all roles are in a good shape, ready to deliver.
Scope definition
Test one role with
moleculeTest multiple roles with
toxAutomate on Continuous Integration platform
Test one role at user request, locally
Before committing any changes to Git (tradeoffs: blocking, slow, antivirus).
Execute all tests on the named role
${rolename}:
cd ${rolename}
molecule test --allTox
Orchestrate tests on a collection of roles.
Isolated python virtual environments.
tox aims to automate and standardize testing in Python. It is part of a larger vision of easing the packaging, testing and release process of Python software.
— https://tox.readthedocs.io
pip install tox virtualenvConfigure tox.ini
Molecule role == Tox named environment
Running platforms are not isolated!
[tox]
envlist =
my_example_role
another_role
skipsdist = true
[testenv]
basepython = python3
commands = bash -c "(cd {toxinidir}/roles/{envname} && molecule --debug
test --all)"
description = molecule test role {envname}
deps = -r {toxinidir}/requirements.txt
setenv = MOLECULE_EPHEMERAL_DIRECTORY={envname}
sitepackages = true
whitelist_externals =
/bin/bash
/usr/bin/rubocopExecute tox tests
tox (1)
tox -e ${rolename} (2)| 1 | Execute all molecule tests on all roles |
| 2 | Same, limited to the named role ${rolename} |
Automation on a continuous integration server
On pull request. Never merge broken code!
At commit on release branches / tags.
Foresee long compute time !
Need to run privileged Docker containers
Docker socket needed!
We run molecule from a docker container, and from this docker container we need to create other container: we need access to the docker socket!
Current implementation of docker support in Bamboo doesn’t allow this (unsafe) behaviour.
We had to bake our own docker orchestration sequence (handled by ansible/tests/tests_suite.sh)
Orchestration pitfalls aka. the monorepo nightmare
We have one unique Git repository containing the complete collection of all roles. It means that when CI runs the tests, all roles should be tested. We would avoid a lot of complication if each role would have its own repository. It is a bad architectural choice, but a business decision and we have to live with it.
Tox is able to run the tests in parallel, instead of sequentially. But blind parallelization on one unique agent is a bad idea our usecase.
To test the elasticsearch and redis roles for instance, we create complex topologies which use a lot of memory. As a consequence we need to fine tune the orchestration to avoid out of memory errors when running to many memory hungry tests in parallel.
Given on CI platform (bamboo), the best approach we found was to group roles in multiple test suites, each suite being run on a different agent (handled by ansible/tests/detox_wrapper.sh).
all elasticsearch related roles, to test sequentially
all redis related roles, to test sequentially
all other linux related roles, candidates to maximize parallelism
all azure/windows bound roles, also candidates to maximize parallelism (and subject to high error rate)
Implementation examples
Kubernetes Operator SDK
geerlingguy.phpmyadmin
Create our own
Kubernetes Operator SDK
Prerequisites
git
docker version 17.03+.
kubectl version v1.9.0+.
ansible version v2.6.0+
ansible-runner version v1.1.0+
ansible-runner-http version v1.0.0+
dep version v0.5.0+.
go version v1.10+.
Access to a Kubernetes v.1.9.0+ cluster.
Install Operator SDK
export GOPATH=$HOME/.go
mkdir -p $GOPATH/src/github.com/operator-framework
cd $GOPATH/src/github.com/operator-framework
git clone https://github.com/operator-framework/operator-sdk
cd operator-sdk
git checkout master
make dep
make installCreate new project
operator-sdk new memcached-operator --api-version=cache.example.com/v1alpha1 --kind=Memcached --type=ansible
cd memcached-operatorMolecule test
Phpmyadmin role from Galaxy
mkvirtualenv molecule
pip install molecule docker
git clone git@github.com:geerlingguy/ansible-role-phpmyadmin.git geerlingguy.phpmyadmin
cd ansible-role-phpmyadmin.git
molecule testCreate a new role
Demo time
Molecule community update
ansible-lint and molecule are great tools. They’ve been built and tested by the community that we see as essential parts of enhancing development of Ansible automation. By adopting these tools, Red Hat intends to invest resources working with the community to make them even better.
2018-09-26
— @tima (Timothy Appnel)
— @tima (Timothy Appnel)
Molecule working group
Continuous integration
Pre release 2.20a2
Ansible Collections
We want molecule to become the defacto standard tool, or sdk if you will, for content creators.
2019-03-13
— @thaumos (Dylan Sylva)
— @thaumos (Dylan Sylva)